CSI-SOP-CON-001 — Container Patch Deployment & Rollback

Purpose

Execute container image patches safely and consistently under CSI-STD-CON-001.

Scope

Docker Compose workloads managed by CSI on Linux hosts. Windows containers and Kubernetes require platform-specific annexes.

Roles

• Change owner: prepares scope, risk, evidence and approvals.
• Implementer: performs backup, deployment and rollback.
• Validator: checks technical and business health. Tier 1 validation must be independent.
• Approver: authorises the maintenance window and residual risk.

Entry Conditions

Do not begin until all are true:

• [ ] Client, site, host and stack are positively identified.
• [ ] Current Compose/manifests are committed to protected Git.
• [ ] Current and proposed image tags and SHA-256 digests are recorded.
• [ ] Release notes, licence changes and breaking changes are reviewed.
• [ ] Exact proposed digest has passed vulnerability assessment.
• [ ] Representative lab tests passed.
• [ ] Backup completed and access to recovery material verified.
• [ ] Rollback trigger, previous digest and commands are recorded.
• [ ] Change approval and maintenance window are active.

Safe Command Setup

Run from the approved stack directory. Replace placeholder values deliberately.

set -euo pipefail

STACK_DIR="/srv/csi/stacks/REPLACE_STACK"
COMPOSE_FILE="$STACK_DIR/compose.yaml"
CHANGE_ID="REPLACE_CHANGE_ID"
APPROVED_IMAGE="vendor/application:1.2.3@sha256:REPLACE_WITH_APPROVED_DIGEST"

test "$STACK_DIR" != "/srv/csi/stacks/REPLACE_STACK"
test "$CHANGE_ID" != "REPLACE_CHANGE_ID"
test "$APPROVED_IMAGE" != "vendor/application:1.2.3@sha256:REPLACE_WITH_APPROVED_DIGEST"
test -d "$STACK_DIR"
test -f "$COMPOSE_FILE"
command -v docker >/dev/null
command -v git >/dev/null
docker compose version >/dev/null
cd "$STACK_DIR"

If any precondition fails, stop and correct the change record.

Procedure

1. Capture Pre-change State

mkdir -p "$STACK_DIR/change-evidence/$CHANGE_ID"
docker compose -f "$COMPOSE_FILE" config   > "$STACK_DIR/change-evidence/$CHANGE_ID/compose-rendered-before.yaml"
docker compose -f "$COMPOSE_FILE" ps   > "$STACK_DIR/change-evidence/$CHANGE_ID/containers-before.txt"
docker image ls --digests   > "$STACK_DIR/change-evidence/$CHANGE_ID/images-before.txt"
git status --short
git rev-parse HEAD   > "$STACK_DIR/change-evidence/$CHANGE_ID/git-commit-before.txt"

Stop if Git shows unexplained changes.

2. Confirm Backup

• Confirm application-consistent backup completion.
• Record backup job ID, timestamp, repository and retention.
• For Tier 1, confirm the latest successful restore test and recovery contact.
• Never store credentials or private keys in the change evidence directory.

3. Validate Proposed Manifest

The reviewed manifest must use an explicit version and approved digest.

services:
  app:
    image: vendor/application:1.2.3@sha256:REPLACE_WITH_APPROVED_DIGEST

docker compose -f "$COMPOSE_FILE" config --quiet
git diff --check
git diff -- "$COMPOSE_FILE"

Compare the diff with the approved change. Stop on any unrelated alteration.

4. Pull Without Deployment

docker compose -f "$COMPOSE_FILE" pull
docker image inspect "$APPROVED_IMAGE" --format '{{json .RepoDigests}}'

Verify the returned digest matches the approved record.

5. Deploy

docker compose -f "$COMPOSE_FILE" up -d --remove-orphans
docker compose -f "$COMPOSE_FILE" ps

Do not run image prune during the observation period.

6. Technical Validation

docker compose -f "$COMPOSE_FILE" ps
docker compose -f "$COMPOSE_FILE" logs --since 15m --no-color \
  > "$STACK_DIR/change-evidence/$CHANGE_ID/logs-after.txt"
mapfile -t CONTAINER_IDS < <(docker compose -f "$COMPOSE_FILE" ps -q)
test "${#CONTAINER_IDS[@]}" -gt 0
docker inspect "${CONTAINER_IDS[@]}" \
  --format '{{.Name}} status={{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{else}}not-defined{{end}} restart={{.RestartCount}}'

Also verify:

• [ ] Health checks are healthy or documented as not implemented.
• [ ] Expected ports and TLS endpoints respond.
• [ ] Authentication and authorisation work.
• [ ] Persistent data is present and writable where expected.
• [ ] Database, cache, proxy and external integrations work.
• [ ] Monitoring and backup jobs remain operational.
• [ ] Defined business transaction succeeds.

7. Observation

• Tier 1: minimum 60 minutes unless a stricter service plan applies.
• Tier 2: minimum 30 minutes.
• Tier 3: minimum 15 minutes.

Record alerts, errors, latency, restart count and client confirmation.

Rollback

Trigger rollback on failed health checks, data-integrity concern, repeated restarts, critical regression, security control failure or approved outage limit.

Image and Manifest Rollback

Restore the approved prior Git revision or exact prior manifest. Do not guess a tag.

PREVIOUS_COMMIT="REPLACE_APPROVED_PREVIOUS_COMMIT"
test "$PREVIOUS_COMMIT" != "REPLACE_APPROVED_PREVIOUS_COMMIT"
git cat-file -e "$PREVIOUS_COMMIT^{commit}"
git show "$PREVIOUS_COMMIT:compose.yaml" > compose.rollback.yaml
docker compose -f compose.rollback.yaml config --quiet
docker compose -f compose.rollback.yaml pull
docker compose -f compose.rollback.yaml up -d --remove-orphans
docker compose -f compose.rollback.yaml ps

The change owner must verify that the file path in git show matches the repository layout before execution.

Data Rollback

If the update changed a schema or data format:

1. Stop writes.
2. Follow the application recovery plan.
3. Restore the recorded application-consistent backup or execute the vendor-supported down-migration.
4. Reconcile transactions created after the recovery point.
5. Validate data integrity before reopening service.

Never restore data merely because the image rollback failed; confirm the recovery decision and its data-loss impact.

Closure

• [ ] Final Compose configuration and deployed digest recorded.
• [ ] Validation and observation evidence attached.
• [ ] Monitoring is normal.
• [ ] Client/change approver receives result.
• [ ] Git repository reflects production.
• [ ] Previous image retained until rollback closure.
• [ ] Failed changes create a problem record.
• [ ] Change record is signed off.

Emergency Variation

Emergency work may abbreviate lab testing but still requires authority approval, verified backup, exact digest identity, focused validation, ready rollback and a completed record within one business day.

Prohibited Actions

• Deploying latest or another floating tag.
• Editing production only through Portainer without updating Git.
• Running unattended pull and up -d schedules.
• Pruning the previous image before change closure.
• Continuing after an unexplained precondition or validation failure.

Related Records

• CSI-STD-CON-001 — Container Patch Management Standard
• IT Change Request & Approval Record

Revision History